Leveraging a rolling cross-section design in panel studies to investigate police legitimacy

Today

Motivation
‘Unexpected event during survey’ design
The rolling cross-section design
This Study: a rolling cross-section panel design
Implementation challenges

Motivation

Confirmatory research

Inferences about changes in police legitimacy
Panel study
- Interview individuals at two or more points in time

Exploratory research

Unexpected events relevant to police legitimacy
Rolling cross-sectional design (RCS)
- Randomized controlled interviewing over time

‘Unexpected event during survey’ design

“research design that exploits the occurrence of an unexpected event during the fieldwork of a public opinion survey to estimate its causal effect on a relevant outcome by comparing responses of the individuals interviewed before the event \(t_i<t_e\) (control group) to those of respondents interviewed after the event \(t_i>t_e\) (treatment group).” (Muñoz et al., 2020, p. 187)

Quasi-experiment: Effects of naturally occurring events that cannot be manipulated through controlled experiments
Assumption that interview time is independent from the time when the event occurs

Unexpected police-related events

The RCS design

An ‘expected-unexpected’ event during survey design

“the design is just a standard cross-section, but the day on which a respondent is interviewed is chosen randomly” (Johnston & Brady, 2022, p. 283)

“The data collected in an RCS allow a fine-grained analysis of dynamic phenomena of public opinion” (Lutz et al., 2013, p. 3)

Temporal heterogeneity as an opportunity to move closer to causal inference (Johnston & Brady, 2002)

The RCS design

The ‘expected-unexpected’ event during survey design

Steps (Kenski et al., 2010):

Total number of interviews is established
Full sample is randomly divided into multiple replicates
Replicates assigned to be interviewed at different intervals of times
The same protocol is followed for all replicates

The RCS design

Example

The RCS design

Assumptions

1. Temporal ignorability: Moment of the interview should be as good as random

Excludability: Timing of the interview \(t\) affects \(Y\) only through event \(T\)
Compliance: assignment to the treatment (\(t_i>t_e\)) corresponds to actually receiving treatment (\(T=1\))

Temporal ignorability: “for any individual, the potential outcomes must be independent from the moment of the interview” (Muñoz et al., p. 189)

Moment of the interview should be as good as random
In RCS moment of interview is random
Protocol to ensure equal procedures to contact and recontact sample units
Need to compare ‘treated’ and ‘not-treated’ in terms of ease of contact and socio-demographics (Lutz et al., 2013)
Assumptions under which unexpected events can yield valid causal estimates
Assignment to different values of time t should be independent from the potential outcomes of Y.
The daily samples need to fulfill two compelling properties: (1) They must consist of a somewhat stable mix of respondents that are easy and such that are harder to reach. (2) The socio-demographic characteristics of their members must not vary significantly over the course of the campaign.(Lutz et al., 2013)
Threats often arise from complex sampling designs. Primary sampling units are often contacted and interviewed sequentially. For example, in face-to-face surveys the fieldwork often follows a geographical pattern for efficiency reasons.
Reachability: ease of contact and reluctance to participate can vary between respondents. In ESS there is a relationship between number of attempted contacts and how late they are interviewed during fieldwork. Therefore, the subjects who are more elusive and harder to reach will have a higher probability of being interviewed after any unexpected event. If reachability is correlated with the potential outcomes, its correlation with treatment assignment will bias the estimates of the causal effect. As expected, those who were older, less educated, and out of the labor market were interviewed earlier during the fieldwork of the ESS survey. This happened less so in RCS designs.
- Should be less of a concern in RCS because day is random. If days overlap, in the long term the design should lead to equal numbers of first/second/third time respondents (Johnston and Brady, 2002).
Attrition: Troublesome if nonresponse is triggered by the treatment. Being exposed to certain events might lead people to refrain from answering some questions or altogether.

Excludability: “any difference between respondents interviewed before and after the event shall be the sole consequence of the event” (Muñoz et al., p. 189)

Timing of the interview \(t\) affects \(Y\) only through event \(T\)
Beware of collateral events, simultaneous events and unrelated time trends
Need to characterize the event and its reactions
The excludability assumption will be violated if the decision about the moment when the event occurs is endogenous to the outcome variable Y.
Collateral events: succession of reactions triggered by the unexpected event of interest. E.g. reactions to a terrorist attack might drive public opinion’s response. Should be interpreted as the joint effect of the envent and the subsequent reactions
Simultaneous events: unrelated events that take place at the same time. Problem of compound treatments. Assumption that out of all the treatments that covary with the timing of the interview, the events of interest is the only treatment that affects the potential outcomes.
Unrelated time trends: presence of time-varying variables that, aside from the event itself, are systematically related to the outcome (temporal stability assumption). E.g. calendar effects

Compliance: “Assignment to the treatment group \(t_i>t_e\) was assumed to perfectly correspond to actually receiving treatment \(T=1\).” (Muñoz et al., 2020, p. 194)

Need to measure compliance
Some of the subjects were not really exposed to the event

This Study

Police legitimacy

Police work requires public support to be able to function in a democratic context
Beliefs that the police act in procedurally just ways and are legitimate facilitate compliance and cooperation with authorities
A socially legitimate police force should require little violence to confront opposition
However, the world has seen many cases of violent demonstrations, police violence and human rights violations

This Study

The Chilean case

Chile’s main police force, the Carabineros, was among the most trusted institutions (Dammert, 2019)
Yet, recent events of corruption and excessive use of force during the social uprising in October 2019 have produced a serious loss of trust (Gerber et al., 2023)
Crisis of legitimacy: proposals of “deep reform” and even the dissolution of the institution
Need to understand relationship between police and citizens in Chile during the next crucial years

This Study

Aims

Confirmatory:
- Track over time people’s perceived police legitimacy and motivational postures towards the police, and examine their relationships with perceived procedural justice and fear of the police in Chile
Exploratory:
- Determine the effects of potential unexpected events (such as police violence, corruption scandals or reforms) on perceptions of legitimacy, procedural justice and fear of the police in Chile.

This Study

A RCS panel design

Probability-based online panel of the population in Chile (18-65 years old)
Panel study with 10 waves
Face-to-face recruitment with probability-based sampling design
Sample size
- Expected sample in contact survey: n = 5,300
- Expected sample in Wave 1: n = 1,500
Possibility to combine with non-probabilistic sample to increase sample size

This Study

A RCS panel design

This Study

A RCS panel design

This Study

A RCS panel design

This Study

Identify unexpected event with RCS design

This Study

Identify unexpected event with panel design

This Study

Identify unexpected events with RCS panel design

Contribution of the RCS panel design

Facilitates the causal identification of unexpected events
- During waves
- Between waves
Potential to identify interactions between
- Unexpected events within and between waves
- Unexpected events with main study hypotheses

But at what cost?

Implementation challenges

Event happens on...	Event		Power
Event happens on...	Not treated	Treated	Small effect	Medium effect	Large effect
Day 4	129	1329	0.58	1.00	1.00
Day 7	257	1200	0.83	1.00	1.00
Day 10	386	1071	0.92	1.00	1.00
Day 13	514	943	0.95	1.00	1.00
Day 17	686	771	0.97	1.00	1.00

Implementation challenges

Thank you!

Contact:

Monica Gerber monica.gerber@udp.cl
Cristóbal Moya cristoba.moya@uni-bielefeld.de